Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic ADD COLUMN and DROP COLUMN replay support for PG, BQ and SF #368

Merged
merged 7 commits into from
Sep 6, 2023

Conversation

heavycrystal
Copy link
Contributor

No description provided.

flow/connectors/postgres/postgres.go Outdated Show resolved Hide resolved
flow/connectors/postgres/postgres.go Outdated Show resolved Hide resolved
flow/connectors/postgres/postgres.go Outdated Show resolved Hide resolved
flow/connectors/snowflake/snowflake.go Outdated Show resolved Hide resolved
flow/connectors/snowflake/snowflake.go Outdated Show resolved Hide resolved
flow/connectors/bigquery/bigquery.go Outdated Show resolved Hide resolved
flow/activities/flowable.go Outdated Show resolved Hide resolved
flow/model/model.go Outdated Show resolved Hide resolved
flow/activities/flowable.go Show resolved Hide resolved
flow/connectors/eventhub/eventhub.go Outdated Show resolved Hide resolved
flow/connectors/s3/s3.go Outdated Show resolved Hide resolved
flow/connectors/postgres/postgres.go Outdated Show resolved Hide resolved
flow/e2e/snowflake/qrep_flow_sf_test.go Show resolved Hide resolved
flow/e2e/bigquery/qrep_flow_bq_test.go Show resolved Hide resolved
@iskakaushik iskakaushik changed the title DRAFT: basic ADD COLUMN and DROP COLUMN replay support for PG, BQ and SF Basic ADD COLUMN and DROP COLUMN replay support for PG, BQ and SF Sep 6, 2023
@iskakaushik iskakaushik merged commit 40a454a into main Sep 6, 2023
iskakaushik added a commit that referenced this pull request Sep 30, 2023
This PR is debugging an issue where schema change propagation (#368)
don't apply properly when there are parallel transactions happening
during an ALTER TABLE statement.

Borrowing from jackc/pglogrepl#59

This PR changes the handling of clientXLogPos to match the one from
Postgres' own `pg_recvlogical`:

- keepalive messages should bump the position too, as they're only sent
(from what I can tell) after any xlogdata message;
- both the WALStart and the ServerWALEnd in logical xlogdata messages
represent the position that should be reported back, and adding the
length of the post-decoding data to it is meaningless;
- relation messages have a position of zero, and in general we should
match the pg_recvlogical behavior of only increasing the local position.

---------

Co-authored-by: Kaushik Iska <[email protected]>
iskakaushik pushed a commit that referenced this pull request Oct 5, 2023
#480)

This PR adds on to the existing work towards syncing schema changes in
CDC mirrors (#368):

1. handles an edge case where a column is dropped and recreated with a
different data type in the same ALTER TABLE statement.
2. Adds quoting, guarding against case-sensitive column names and
reserved identifiers.
3. Fixes type mapping for BigQuery

Additionally, adds some tests for ReplayTableSchemaDelta for Postgres
and Snowflake. Bigquery does not have tests yet but has been manually
tested.
@serprex serprex deleted the schema-changes-basic branch July 19, 2024 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants